The operation of a conventional mechanical or optical pointing or input device such as a mechanical or optical computer mouse is well known in the art. By use of these devices, a user can select files, programs, or actions from lists, groups of icons, etc., and can “gesturally” move files, programs, etc. issue commands or map to specific actions, for example in drawing programs.
As examples, a mechanical computer mouse relies on one or more wheels and/or balls to track movement or displacement information relative to forward-backward and left-to-right movement of the computer mouse, for example by interrupting infrared beams of light directed at light sensors to create pulses representative of wheel or ball movement. Simple logic circuits interpret the relative timing of the pulses to indicate which direction the wheel(s) or ball(s) is moving, which is then converted by driver software into motion of a visual indicator such as a pointer, cursor, or cross-hair along X and Y axes of a computing device display screen.
An optical computer mouse replaces the mechanical mouse wheels or balls with one or more visible or invisible light sources such as light-emitting diodes (LEDs), laser diodes, infra-red light, etc. to detect movement of the mouse relative to an underlying surface such as a mouse pad. The inertial/gyroscopic computer mouse uses a tuning fork or other accelerometer to detect rotary movement for every axis supported, most commonly using 2 degrees of rotational freedom and being insensitive to spatial translation. The user need only perform small wrist rotations to move a pointer or cursor on a display screen.
Almost all modern 2D pointing devices utilize an active approach for detecting movement of the device. The underlying technology of modern surface-independent (meaning that a specific surface type is not required, although some type of surface is) 2D pointing/input devices such as optical mice (see FIG. 1) is known as digital image correlation. The mouse includes a shell or housing and a variety of input means such as left and right buttons, a scroll wheel, etc. A displacement detection LED disposed on a bottom surface of the mouse is used to detect movement of the mouse over a surface. An optical or optoelectronic mouse uses an optoelectronic sensor (essentially, a tiny low-resolution video camera) to image a naturally occurring texture of an underlying surface made of materials such as wood, cloth, mouse pad materials and Formica, using light reflected from the surface. These surfaces, when lit at a grazing angle by the displacement detection LED, cast distinct shadows that resemble a hilly terrain lit at sunset. Images of these surfaces are captured in continuous succession as the mouse is translated over the surface, often at a speed of more than one thousand frames per second. Depending on the speed with which the mouse is moved, each image will be offset from the previous image by a fraction of a pixel or as many as several pixels. By using cross correlation to calculate how much each successive image is offset from the previous image, a displacement processor can determine the distance the mouse has moved from image data captured by the sensor. The movement of the mouse can then be translated or converted into a corresponding movement of a visible marker such as a cursor on a graphical user interface such as a computer screen.
Even though a special surface such as a mouse-pad is not needed by a modern optical mouse, operation of the mouse still requires a surface on which to rest and over which to travel. If a suitable operating surface is not available and an alternative such as a touchpad or trackball is also not available, a conventional optical mouse cannot be used. In turn, certain tasks often done with pointing/input devices such as a computer mouse are difficult to impossible to accomplish with alternative pointing/input devices such as touchpads or trackballs. For example, use of drawing programs without a computer mouse can be difficult if not impossible. Likewise, tasks such as two-dimensional (2D) or 3D sculpturing or drawing, “flying” in multi-dimensional space (for example, three-dimensional space defined by X, Y, and Z axes) such as during gaming, etc. would be difficult to accomplish using a conventional touchpad, trackball, etc. Still more, personal computers (PCs) are not merely tools for surfing the internet and sending e-mail in the modern world. Increasingly, PCs serve as digital media centers to view photos, listen to music, and watch video clips, films and TV shows. Indeed, notebook, laptop, and desktop computers are rapidly becoming accepted accessories or supplements to (and in certain situations replacing) the home entertainment center.
Likewise, the modern television is no longer just a TV, offering integrated Internet capabilities and set-top boxes that offer more advanced computing ability and connectivity than a contemporary basic TV set. The modern “smart” TV can deliver content from computers or network attached storage devices, such as photos, movies and music. These devices also provide access to Internet-based services including traditional broadcast TV channels, pre-recorded programs, video-on-demand, electronic program guides, interactive advertising, personalization, voting, games, social networking, and other multimedia applications. All of these require a remote control-like device that can provide cursor control, i.e. a pointing/input device as is known for computing devices. Unfortunately, traditional television remote controls cannot conveniently provide such functionality. As noted above, a conventional pointing/input device such as a computer mouse requires a desk or other hard surface to function.
For this reason, attempts have been made to adapt the familiar computer mouse to operate in the air or “on the fly,” to avoid the need for a surface over which to translate the mouse for operation. Indeed, 3D pointing has long been a desired feature in human-computer interaction to allow tasks that are not possible with a 2D pointing device, such as 3D sculpturing or space navigating. However, 3D pointing technology has not reached a stage that is considered both reasonably affordable and manipulative.
For 3D pointing, it is necessary to identify the location of the pointing device with respect to a reference point in a 3D space. Unlike 2D pointing which mainly uses an active approach, 3D pointing has been attempted using both active and passive approaches. The approach taken depends on whether the pointing device includes a displacement detection system that works in 3D space. The optical displacement detection system of an optical mouse can only work on a surface due to the operating mechanism summarized above; it cannot work if suspended in 3D space.
In the active approach, typically an imager such as an IR camera is integrated into the pointing device to detect lights from an IR emitter of a console such as the console of a gaming device, and calculate spatial coordinates for the pointing device accordingly. The Wii® Remote marketed by Nintendo® falls within that category. A problem with this approach is that the pointing device spatial coordinates can only be calculated when its imager has a direct line of sight to a sensor bar associated with the gaming device console.
Another active type of 3D mouse uses a tuning fork or other accelerometer to detect rotary movement for every axis supported. Logitech® and Gyration's inertial mice (also called gyroscopic mice) fall in this category. The most common models work using 2 degrees of rotational freedom. An operator uses wrist rotations to move the cursor. The inertial mouse is insensitive to spatial translations. More recently, an inertial mouse was developed, equipped with g-sensors (pairs of accelerometers extended over a region of space) to calculate the mouse position, orientation and velocity; hence such a mouse can provide at least 9 spatial parameters for pointing purposes. However, the price of such an inertial mouse is quite high; usually 10 times more than the price of a typical optical mouse.
For pointing devices that do not include a distance measuring component, a passive approach has been evaluated requiring a separate component to measure the distance between the pointing device and, for example, a gaming device or base station, or to identify the location of the pointing device with respect to the gaming device or base station. All gesture-based pointing device approaches, such as the Kinect® device marketed by Microsoft®, belong to this latter category. In this case, the fingers or the hands of a user play the role of a pointing device and a special imaging device is required to identify the locations of the fingers or hands of the user. Three-dimensional mice such as 3Dconnexion/Logitech's® SpaceMouse® in the early 1990s and Kantek's® 3D RingMouse® in the late 1990s, also known as bats, flying mice or wands, also fall in this category. As an example, the RingMouse® was tracked by a base station through ultrasound. This approach has been found to provide insufficient resolution.
Still other attempts have been made to implement passive detection of a pointing device location by combining pointing and imaging functionalities in a single device. In one such device, a digital camera mounted into the housing of a computer mouse includes a mode selection system to switch the device between a 2D mouse function and a digital camera function (see FIG. 1). The device could not be used for 3D pointing. In another attempt, three one-dimensional imaging devices were used to track a point light source, to recognize and execute gestures input by the operator. The problem with this approach is that a holding time is required before each operation such as a click or drag can be performed.
To date, the present inventors are unaware of any attempts to use a single-lens imaging device to capture the motion and clicking activities of an “on the fly” pointing device for 2D and 3D pointing in a 3D human-computer interaction system.